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INTRODUCTION 


The  Department  of  Defense  is  exponentially  inereasing  the  aequisition  of  joint  eomplex 
systems  that  deliver  needed  eapabilities  demanded  by  our  warfighter.  Systems  engineering  is 
the  technical  and  technical  management  process  that  focuses  explicitly  on  delivering  and 
sustaining  robust,  high-quality,  affordable  solutions.  Air  Force  leadership  has  collectively 
stated  the  need  to  mature  a  sound  systems  engineering  process  throughout  the  Air  Force. 
Gaining  an  understanding  of  the  past  and  distilling  lessons  learned  that  are  then  shared  with 
others  through  formal  education  and  practitioner  support  are  critical  to  achieving  continuous 
improvement. 

This  synopsis  conveys  the  salient  results  of  case  studies  focused  on  the  application  of 
systems  engineering  principles  within  various  programs.  Salient  results  are  conveyed  as 
learning  principles  to  facilitate  pedagogy.  But  these  results  are  also  useful  to  practicing 
engineers  and  managers  as  they  apply  systems  engineering  throughout  a  weapon  system’s 
life  cycle.  The  reader  is  encouraged  to  delve  into  the  details  contained  in  the  complete  case 
study  should  a  particular  learning  principle  relate  to  a  situation  on  your  program. 

Each  learning  principle  is  identified  as  follows: 

(short  name)  /  (learning  principle  number) 

The  short  name  key  is  as  follows: 

F-1 1 1  refers  to  the  F-1 1 1  Systems  Engineering  Case  Study 

B-2  refers  to  the  B-2  Systems  Engineering  Case  Study 

C-5  refers  to  the  C-5A  Galaxy  Systems  Engineering  Case  Study 

GPS  refers  to  the  Global  Positioning  System  Systems  Engineering  Case  Study 

HST  refers  to  the  Hubble  Space  Telescope  Systems  Engineering  Case  Study 

TBMCS  refers  to  the  Theater  Battle  Management  Core  System  Systems  Engineering  Case 

Study 

Peacekeeper  refers  to  the  Peacekeeper  Intercontinental  Ballistic  Missile  Systems  Engineering 
Case  Study 

A- 10  refers  to  the  A- 10  Thunderbolt  II  (Warthog)  Systems  Engineering  Case  Study 
GH  refers  to  the  Global  Hawk  Systems  Engineering  Case  Study 
KC-135  refers  to  the  KC-135  Simulator  Systems  Engineering  Case  Study 

The  learning  principle  title  is  highlighted  green  if  it  contains  information  related  to  an 
application  of  systems  engineering  that  one  should  consider  adopting.  The  learning  principle 
title  is  highlighted  yellow  if  it  contains  information  related  to  problems  that  should  be 
avoided  in  the  application  of  systems  engineering. 

Complete  case  studies  are  available  on  the  Air  Eorce  Center  for  Systems  Engineering  website 
at  [http://www.afit.edu/cse]. 
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CASE  STUDY  LEARNING  PRINCIPLES 


B-2/1  Integration  of  the  Requirements  and  Design  Process 

A  key  aspect  of  the  implementation  of  the  B-2  system  engineering  process  was  the 
integration  of  the  SPO  requirement’s  team  with  the  contractor’s  design  team,  including 
manufacturing,  Quality  Assurance,  and  logistics  functionals  into  a  cohesive  program  effort. 
This  facilitated  continual  trade  studies  conducted  by  the  specialists  from  the  User/SPO 
government  team  with  the  company  specialists  to  fully  assess  the  performance  trade-offs 
against  schedule,  cost,  and  risk. 

B-2/2  WBS  Task  Teams  and  Functional  Hierarchy 

The  contract  Work  Breakdown  Structure  (WBS)  stipulated  the  entire  program  content  and 
tasking  and  the  company  organized  the  development  effort  into  WBS  teams  responsible  to 
implement  the  contract  WBS.  These  WBS  Task  Teams  were  assigned  complete  work 
packages  -  for  example,  the  forward  center  wing.  The  systems  engineering  WBS  Task  Team 
efforts  were  organized  similarly,  but  with  separate  responsibilities,  each  reporting  the 
Northrop  chief  engineer  or  his  deputies.  The  functional  organizations  assigned  members  to 
the  task  teams  to  assure  accommodation  of  their  program  needs.  A  vital  distinction  from 
many  of  today’s  IPTs  was  retaining  the  WBS  Task  Team  membership  throughout  the 
functional  organizations’  various  management  levels.  This  facilitated  communication, 
integration,  interfaces,  and  integrated  the  functional  leadership  of  each  of  the  company’s 
technical  and  management  disciplines  into  the  decision  process.  The  program  management 
top  level  structure  was  organized  into  a  strong  project  office  with  centralized  decision 
authority  and  strong  leadership  at  the  top  of  both  the  SPO  and  the  contractor  organizations. 

B-2/3  Air  Vehicle  Reconfiguration 

The  identification  of  a  major  aeronautical  control  inadequacy  of  the  baseline  configuration 
just  four  months  prior  to  the  formal  Configuration  Freeze  milestone  caused  an  immediate 
refocus  of  the  Task  Teams  to  develop  a  substantially  revised  design.  Within  several  days,  the 
air  vehicle  task  teams  were  conducting  trade  studies,  augmenting  their  skill  sets,  and 
integrating  with  the  other  program  participants  in  a  coordinated  effort  to  derive  an  efficient, 
controllable,  operationally  useful  system.  At  the  same  time,  the  program  elements  that  were 
not  markedly  affected  by  the  change  maintained  a  course  that  preserved  their  schedule,  but 
was  sufficiently  flexible  to  include  any  potential  changes.  In  a  program  wide  systems 
engineering  effort,  the  prime  contractor’s  program  office  integrated  the  teams,  reviewed  their 
efforts,  coordinated  the  systems  trades,  and  identified  significant  changes  to  the  outer  mold 
lines,  the  radar  cross  sections  (RCS)  baseline,  all  major  structure  assemblies,  and  all  major 
air  vehicle  subsystems  requirements,  with  the  exception  of  avionics  and  armament.  The 
alternatives  were  derived  by  the  end  of  the  third  month,  the  final  choice  was  selected  by  the 
sixth  month,  and  the  seventh  month  was  used  to  coordinate  and  garner  the  approval  of  all 
stakeholders.  While  the  program  response  to  the  crises  was  rapid  and  effective,  and  a 
significant  impact  on  the  downstream  cost  and  schedule  was  anticipated  by  the  management 
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team,  and  the  technical  impact  was  predicted  by  the  systems  engineering  process,  it  was  not 
predicted  to  the  fullest  extent. 

B-2/4  Subsystem  Maturity 

The  effect  of  the  reconfiguration  on  the  maturity  of  all  the  air  yehicle  subsystems  (flight 
control,  enyironmental  control,  electrical,  landing  gear,  etc)  was  far  greater  than  projected. 
The  subsystems  were  mostly  yendor-supplied  equipments  and  some  were  in  the  selection 
process  to  the  technical  requirements  of  the  original  baseline  when  the  reconfiguration 
occurred.  After  the  new  configuration  was  deriyed,  the  requirements  for  the  subsystems 
changed  to  such  a  degree  that  they  had  to  be  resized  and  repackaged.  It  took  longer  than 
anticipated  by  the  systems  engineering  process  to  recognize  the  growing  problem  of  getting 
all  the  specifications  updated  and  to  identify  the  lagging  equipment  maturity  that  resulted. 
Thus,  the  reconfiguration  required  a  second  iteration  of  the  design  requirements  and  their 
flow-down  to  the  many  suppliers  and  their  detailed  designs.  These  iterations  after  PDR-2 
resulted  in  the  yehicle  subsystems  not  achieying  their  Critical  Design  Reyiew  (CDR) 
milestone  concurrently  with  the  structure,  but  rather  fiye  months  later. 

B-2/5  Risk  Planning  and  Management 

The  program  was  structured  so  that  all  risks  affecting  the  yiability  of  the  weapons  system 
concept  were  identified  at  contract  award  and  were  structured  as  part  of  the  Program  and 
WBS  work  plans.  The  initial  risks  were  comprised  of  those  “normal”  risks  associated  with  a 
large  complex  weapons  system  deyelopment,  as  well  as  the  new  technology  and  processes 
necessary  to  mature  the  program  to  low  to  medium  risk  at  PDR.  Those  initial  risks  were 
closed  prior  to  PDR  2.  The  risk  closure  process  continued  throughout  deyelopment  and 
identified  new  risks  and  continuously  identified  new  risk  closure  plans.  Most  importantly, 
the  work  associated  with  risk  closure  for  each  plan  was  integrated  into  the  WBS  task  teams’ 
work  plans  and  into  the  Program  Plans.  These  detailed  plans  showed  all  design,  analyses, 
tests,  tooling,  and  other  tasks  necessary  to  close  the  identified  risks  and  were  maintained  and 
reyiewed  as  part  of  the  normal  design/program  reporting  actiyity. 

C-5/1  Requirements 

The  process  for  deyeloping  and  documenting  the  system  performance  requirements 
integrated  the  User  (warfighter),  planners,  deyelopers,  and  technologists  from  both  the 
goyemment  and  industry  in  a  coordinated  set  of  trade  studies.  It  resulted  in  a  well-balanced, 
well-understood  set  of  requirements  that  fundamentally  remained  unchanged  throughout  the 
program. 

C-5/2  Total  Package  Procurement  Concept  (TPPC) 

The  Total  Package  Procurement  Concept  (TPPC)  employed  by  the  goyemment  required  a 
fixed-price,  incentiye  fee  contract  for  the  design,  deyelopment,  and  production  of  58  aircraft. 
It  included  a  clause  giying  Total  Systems  Performance  Responsibility  (TSPR)  to  the  prime 
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contractor.  TPPC  was  invented  to  control  costs,  but  it  was  the  underlying  cause  of  the  cost 
overrun  and  limited  number  of  aircraft  purchased  under  the  original  contract. 

C-5/3  Weight  Empty  Guarantee 

A  Weight  Empty  Guarantee  was  included  in  the  specification  as  a  performance  requirement 
and  in  the  contract  as  a  cost  penalty  for  overweight  conditions  of  delivered  aircraft.  The 
aircraft  Weight  Empty  Guarantee  dominated  the  traditional  aircraft  performance 
requirements  (range,  payload,  etc.),  increased  costs,  and  resulted  in  a  major  shortfall  in  the 
wing  and  pylon  fatigue  life.  The  stipulation  of  a  Weight  Empty  Guarantee  as  a  performance 
requirement  had  far-reaching  and  significantly  deleterious  unintended  consequences. 

C-5/4  Independent  Review  Teams  (IRTs) 

The  Air  Eorce  C-5  Systems  Program  Office  employed  Independent  Review  Teams  (IRTs)  to 
assemble  national  experts  to  examine  the  program  and  provide  recommendations  to  the 
government.  These  problem-solving  teams  were  convened  to  gamer  the  best  advice  in 
particular  technical  areas:  stmcture  design  and  technology,  and  designs  to  achieve  useful 
service  life. 

F-111/1  Requirements  Definition  and  Management 

Ill-conceived,  difficult- to-achi eve  requirements  and  attendant  specifications  made  the  E-111 
system  development  extremely  costly,  risky  and  difficult  to  manage. 

F-1 1 1/2  Systems  Architecture  and  Design  Trade-Offs 

F-1 1 1  Systems  Engineering  managers  (both  government  and  contractor)  were  not  allowed  to 
make  the  important  tradeoffs  that  needed  to  be  made  in  order  to  achieve  an  F-1 1 1  design  that 
was  balanced  for  performance,  cost  and  mission  effectiveness  (including  survivability)  and 
the  attendant  risk  and  schedule  impacts. 

F-1 1 1/3  Communications  and  Systems  Management 

The  F-1 11  suffered  from  poor  communications  between  the  Air  Force  and  Nayy  technical 
staffs,  and  from  oyer-management  by  the  Secretary  of  Defense  and  the  Director,  Defense 
Research  and  Engineering,  and  it  came  under  intense  congressional  scrutiny,  which  restricted 
the  System  Program  Office  (SPO)  Director  from  applying  sound  systems  engineering 
principles. 

F-1 1 1/4  Validation  and  Verification 

The  F-1 1 1,  like  any  complex  weapon  system  deyelopment  program  which  proyides  new  war¬ 
fighting  capability,  had  areas  of  risk  or  deficiency  that  came  to  light  during  RDT&E  eyen 
though  there  was  perceiyed  low  risk  in  the  design.  The  F-1 11  deyelopment  program 
introduced  concurrency  (oyerlap)  between  design  yalidation/yerification  and  production  to 
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accelerate  program  schedules  because  there  was  an  urgent  need  for  the  capability.  However, 
technical  problems  uncovered  in  verification  and  validation  led  to  costly  retrofits  and 
redesign  of  the  production  versions.  The  most  notable  technical  problems  during  the  F-111 
development  were  inlet-engine  compatibility,  structural  failures  in  the  wing  carry-through 
structure,  and  the  introduction  of  the  technically  immature  digital  avionics  system  (call  the 
Mark  II)  to  replace  the  baseline  analog  avionics  system. 

F-1 1 1/5  Program  Management 

Cancellation  of  the  Navy  F-1 1  IB  in  1968,  after  bi-service  design  was  frozen  in  1964  and 
production  of  the  Air  Force  F-1 1  lA  was  well  underway,  had  a  lasting  impact  on  the  United 
States  Air  Force  (USAF)  F-111  performance  and  cost. 


GPS/1  Programs  must  strive  to  staff  key  positions  with  domain  experts. 


From  program  management  to  systems  engineering,  to  design,  to  the  manufacturing  and 
operations  teams,  the  people  on  the  program  were  well-versed  in  their  disciplines,  and  all 
possessed  a  systems  view  of  the  program.  While  communications,  working  relationships, 
and  organization  were  important,  it  was  the  ability  of  the  whole  team  at  all  levels  to 
understand  the  implications  of  their  work  on  the  system  that  was  vital.  Their  knowledge- 
based  approach  for  decision  making  had  the  effect  of  shortening  the  decision  cycle,  because 
both  the  information  was  understood  and  the  base  and  alternative  solutions  were  accurately 
presented. 


GPS/2  The  systems  integrator  must  rigorously  maintain  program  baselines. 


The  JPO  retained  the  role  of  managing  and  controlling  the  system  specification  and, 
therefore,  the  functional  baseline.  The  JPO  had  derived  and  constructed  a  mutually  agreed-to 
set  of  system  requirements  that  became  the  program  baseline  in  1973.  While  conducting  the 
development  program,  the  GPS  team  was  able  to  make  performance/risk/cost  trade  analysis 
against  the  functional  baseline  to  control  both  risk  and  cost.  The  JPO  was  fully  cognizant  of 
the  implications  of  the  functional  requirements  on  the  allocated  baseline  because  they 
managed  the  Interface  Control  Working  Group  process.  Managing  that  process  gave  them 
firs-hand  knowledge  and  insight  into  the  risks  at  the  lowest  level. 

GPS/3  Achieving  consistent  and  continuous  high-level  support  and  advocacy  helps  funding 

stability,  whieh  impacts  SE  stability. 

Consistent,  continuous  high-leyel  support  proyided  requirements  and  funding  stability.  In 
this  role,  the  OSD  proyided  adyocacy  and  sourced  the  funding  at  critical  times  in  the 
program,  promoted  coordination  among  the  yarious  seryices,  and  reyiewed  and  approyed  the 
GPS  JPO  system  requirements.  OSD  played  the  central  role  in  the  establishment  and 
suryiyability  of  the  program.  The  GPS  JPO  had  clear  support  from  the  Director  of  Defense 
Deyelopment,  Research  and  Engineering  (DDR&E),  Dr.  Malcolm  Currie,  and  program 
support  from  the  Deputy  Secretary  of  Defense,  Dr.  Dayid  Packard.  Clearly  the  seryices  - 
particularly  the  Nayy  and  the  Air  Eorce  early  on,  and  later  the  Army  -  were  the  primary  users 
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and  the  eventual  customers.  However,  each  service  had  initial  needs  for  their  individual 
programs,  or  for  the  then-current  operational  navigation  systems.  Additionally,  the  Secretary 
of  the  Air  Force  provided  programmatic  support  to  supply  manpower  and  facilities. 

GPS/4  Disciplined  and  appropriate  risk  management  must  be  applied  throughout  the 

lifecycle. 

The  GPS  program  was  structured  to  address  risk  in  several  different  ways  throughout  the 
multiphase  program.  Where  key  risks  were  known  up  front,  the  contractor  and/or 
government  utilized  a  classic  risk  management  approach  to  identify  and  analyze  risk,  and 
developed  and  tracked  mitigation  actions.  These  design  (or  manufacturing/launch)  risks 
were  managed  by  the  office  who  owned  the  risks.  Identified  technical  risks  were  often 
tracked  by  Technical  Performance  Measures  (TPMs),  (e.g.  satellite  weight  and  Software 
Lines  of  Codes  (SLOC)),  and  addressed  at  weekly  chief  engineer’s  meetings. 

HST/1  Early  and  full  participation  by  the  customer/user  throughout  the  program  is  essential 

to  program  success. 

In  the  early  stages  of  the  HST  program  the  mechanism  was  not  well  defined  and  the  user 
community  was  initially  polarized  and  not  effectiyely  engaged  in  program  definition  and 
advocacy.  This  ultimately  changed  for  the  better,  even  if  driven  heavily  by  external  political 
and  related  national  program  initiatives.  Ultimately,  institutionalization  of  the  user’s  process 
for  involvement  ensured  powerful  representation  and  a  fundamental  stake  and  role  in  the 
program  requirements  and  requirements  management.  Over  time,  the  effectiveness  of  “The 
Institute”  led  to  equally  effective  user  involvement  in  the  operational  aspects  of  system 
(deployment  and  operations)  as  well. 

HST/2  The  use  of  Pre-Program  Trade  Studies  (“Phased  Project  Planning”  in  NASA  parlance 

at  the  time)  to  broadly  explore  technical  concepts  and  alternatives  is  essential  and  provides 

for  a  healthy  variety  of  inputs  from  a  variety  of  contractors  and  government  (NASA)  centers. 

These  activities  cover  a  range  of  feasibility,  conceptual,  alternative  and  preliminary  design 
trades  with  cost  initially  a  minor,  then  later  a  major,  factor  as  the  process  proceeds.  For  HST, 
several  Headquarters  and  Center  organizations  funded  these  studies  and  sponsored  technical 
workshops  for  HST  concepts.  This  can  promote  healthy  or  unhealthy  competition,  especially 
when  roles  and  responsibilities  within  and  between  the  participating  management  Centers 
have  not  yet  been  decided  and  competing  external  organizations  use  these  studies  to  further 
both  technical  and  political  agendas.  Center  roles  and  missions  can  also  be  at  stake 
depending  on  political  and  or  budgetary  realities.  The  systems  engineering  challenge  at  this 
stage  is  to  “keep  it  technical,  stupid!” 

HST/3  Provision  for  a  high  degree  of  systems  integration  to  assemble,  test,  deploy  and 

operate  the  system  is  essential  to  success  and  must  be  identified  as  a  fundamental  program 

resource  need  from  early  on  (part  of  the  program  baseline). 


Approved  for  Public  Release;  Distribution  Unlimited 


For  FIST,  the  early  wedding  of  the  program  to  the  Shuttle,  prior  NASA  9and  of  course, 
NASA  contractors)  experience  with  similarly  complex  programs,  such  as  Apollo,  and  the 
early  requirement  for  manned,  on-orbit  servicing  made  it  hard  not  to  recognize  this  was  a  big 
SE  integration  challenge.  Nonetheless,  collaboration  between  government  engineers, 
contractor  engineers,  as  well  as  customers,  must  be  well  defined  and  exercised  early  on  to 
overcome  inevitable  integration  challenges  and  unforeseen  events. 

HST/4  Life  Cycle  Support  Planning  and  Execution  must  be  integral  from  day  one  (including 

concept  and  design  phases)  and  the  results  will  speak  for  themselves.  Programs  structured 

with  real  life  cycle  performance  as  a  design  driver  will  be  capable  of  performing  in-service 

better,  and  will  be  capable  of  dealing  with  unplanned,  unforeseen  events  (even  usage  in 

unanticipated  missions.) 

HST  likely  represents  the  benchmark  for  building-in  system  sustainment  (reliability, 
maintainability,  provision  for  technology  upgrade,  built-in  redundancy,  etc.),  all  with 
provision  for  operational  human  execution  of  functions  (planned  and  unplanned)  critical  to 
servicing  missions.  With  four  successful  service  missions  complete,  including  one  initially 
not  planned  (the  primary  mirror  repair),  the  benefits  of  design-for-sustainment,  or  life  cycle 
support,  throughout  all  phases  of  the  program,  becomes  quite  evident.  Had  this  not  been  the 
case,  it  is  not  likely  that  the  unanticipated,  unplanned  mirror  repair  could  have  even  been 
attempted,  let  alone  be  totally  successful. 

HST/5  Eor  complex  programs,  the  number  of  players  (government  and  contractor)  demands 

that  the  program  be  structured  to  cope  with  high  risk  factors  in  many  management  and 

technical  areas  simultaneously. 

Eor  HST,  there  was  heavy  reliance  on  the  contractors  (especially  Lockheed  (LMSC)  and 
Perkin-Elmer  (P-E)  and  they  each  “owned”  very  significant  and  unique  program  risk  areas. 
In  the  critical  optical  system  area,  and  with  EM  as  the  overall  integrator,  there  was  too  much 
reliance  on  EM  to  manage  risk  in  an  area  where  P-E  was  clearly  the  technical  expert. 
Accordingly,  NASA  relied  on  LMSC  and  LMSC  relied  on  P-E  with  insufficient  checks, 
oversight  and  independence  of  the  QA  function  throughout.  While  most  other  risk  areas 
were  no  doubt  managed  effectively,  lapses  here  led  directly  to  the  primary  mirror  defect 
going  to  orbit  undetected  in  spite  of  substantial  evidence  that  could  have  been  used  to  prevent 
this  occurrence. 

TBMCS/1  Requirements  Definition  and  Management 

The  government  did  not  produce  a  Concept  of  Operations,  key  operational  performance 
parameters,  or  a  system  specification  for  the  contractor.  The  contractor  was  responsible  for 
generating  a  systems  segment  specification  that  had  performance  measures  as  goals,  but  not 
testable  requirements.  The  government  did  produce  a  technical  requirements  document  that 
defined  a  technical  approach  and  levied  certain  standards  on  the  contractor.  There  was  no 
firm  baseline  for  operational  and  system  requirements  form  which  the  system  could  be  built 
and  tested.  The  requirements  baseline  was  volatile  up  to  system  acceptance,  which  took 
place  after  TBMCS  passed  operational  test  and  evaluation. 
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TBMCS/2  System  Architecture 


The  system  architecture  was  defined  at  too  high  a  level,  which  had  a  tremendous  impact  on 
system  design  and  development.  The  government’s  mandates  for  software  reuse  and  use  of 
commercial  software  products  were  often  contradictory  and  problematic  for  the  system 
development.  The  layered  system  architecture  did  support  system  evolution  and  migration  to 
modem  technologies. 

TBMCS/3  System/Subsystem  Design 

The  system  and  subsystem  design  was  severely  hampered  by  the  complexity  of  legacy 
applications,  misunderstanding  of  the  maturity  and  complexity  of  commercial  and  third  party 
software  products,  and  a  lack  of  understanding  of  how  the  system  would  be  employed  by  the 
user. 

TBMCS/4  System  Integration 

Systems  and  interface  integration  was  highly  complex.  The  system  integration  was  very 
difficult  because  of  the  lack  of  detail  in  the  system  architecture  and  the  mandate  to  use 
government-furnished  equipment  that  was  not  necessarily  compatible  with  commercial  off- 
the-shelf  products.  Integrating  third  party  software  products  was  an  arduous  process  and 
required  extensive  oversight.  The  external  system  interfaces  were  not  managed  and  were 
often  impossible  to  test  at  the  contractor’s  facility. 

TBMCS/5  Validation  and  Verification 

The  lack  of  a  firm  requirements  baseline  made  validation  and  verification  very  difficult.  The 
program  was  schedule  driven  and  often  ran  parallel  test  processes  without  clear  measures  of 
success.  Not  being  able  to  replicate  the  operational  environment  prior  to  acceptance  test 
created  severe  problems. 

Peacekeeper/ 1  Development  commands  must  manage  their  technology  base  to  optimize 
progress  over  several  programs. 

During  this  period  of  Intercontinental  Ballistic  Missile  (ICBM)  development,  BMO  was  able 
to  develop  and  manage  a  technology  base  development  plan  that  spanned  several  programs. 
BMO  and  its  predecessors  began  developing  technologies  with  the  Atlas  system  that 
continued  through  the  Titan,  Minuteman,  Peacekeeper  and  Small  ICBM.  This  included 
technologies  such  as  solid  rocket  propellants,  nozzle  manufacture,  liquid  fueled  engines,  and 
guidance  systems.  The  solid  rocket  motors  on  Peacekeeper  were  variants  of  the  design  used 
on  Minuteman.  These  efforts  were  integrated  with  the  Air  Force  laboratories,  support 
contractors,  system  contractors  and  Systems  Engineering/Technical  Assistance  (SE/TA) 
contractors.  Thus,  a  Systems  Program  Office  (SPO)  was  able  to  manage  the  technology 
development  efforts  for  the  next  program  so  that  the  lessons  learned  from  the  previous 
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system  could  be  applied  and  technology  transition  decisions  could  be  made  based  on  an 
intimate  knowledge  of  the  result  of  the  technology  development  efforts. 

Peacekeeper/2  The  systems  integrator  must  maintain  rigorous  control  of  these  processes  and 
all  of  the  supporting  contractors. 

The  USAF  SPO  was  the  lead  integrator  for  the  contractor  team  with  the  support  of  the 
Systems  Engineering/Technical  Assistance  (SE/TA)  contractor  (TRW),  which  enabled  the 
USAF  to  own  the  systems  engineering  process  and  missile  specifications.  The  SPO  also 
served  as  the  configuration  control  authority  for  the  specifications  for  each  associate 
contractor  and  for  the  Interface  Control  Documents  (ICDs)  coordinating  associate  contractor 
deliverables. 

The  systems  engineering  role  was  split  between  the  SPO,  TRW  as  the  SE/TA  Contractor, 
Martin  Marietta  as  the  Assembly  and  Check  Out  (A&CO)  contractor  for  the  missile,  and 
Boeing  as  the  contractor  for  the  basing  equipment  including  A&CO. 

■  The  SPO  provided  contractual  authority  and  oversight  and  controlled  the 
specifications  and  ICDs  for  each  associate  contractor.  The  SE/TA  contractor 
(TRW)  provided  insight  and  advice  to  the  SPO.  The  hands-on  physical 
integration  at  the  test  sites  and  in  the  field  was  handled  by  the  A&CO  contractors. 

■  There  were  many  ramifications  including  significant  SPO  and  SE/TA  manpower 
requirements.  Perhaps  the  most  important  benefit  was  the  ability  of  the 
Government  and  its  SE/TA  support  to  conduct  both  technical  and  business 
tradeoffs  and  then  impose  the  results  contractually  without  concern  for  overriding 
a  prime  contractor  and  the  vastly  more  detailed  insight  into  the  execution  of  the 
program. 

■  As  the  lead,  the  Air  Force  had  the  right  to  overrule  contractors  and  even  to  re¬ 
compete  their  products  and  services  if  required. 


Peacekeeper/3  The  systems  integrator  must  maintain  a  rigorous  systems  engineering  process. 


The  System  Requirements  Analysis  (SRA)  evolved  during  the  previous  Minuteman  program 
and  brought  significant  discipline  to  the  systems  engineering  process  through  four  key 
elements: 

(i)  The  Operational  Requirements  Analysis  (ORA),  which  defined  and  allocated  the 
technical  requirements  for  the  prime  operational  equipment  for  each  associate 
contractor  to  include  Technical  Orders  (TOs),  training,  and  other  activities  which 
also  indicated  the  requirements  for  Strategic  Air  Command  (SAC)  operational 
personnel. 

(ii)  Test  Planning  Analysis  (TP A)  that  developed  the  requirements  and  plans  for 
testing  including  the  Integrated  Test  Plan  (ITP)  by  which  all  developmental  and 
operational  testing  was  planned  leading  up  to  and  including  flight  testing  and  that 
developed  the  requirements  for  test  facilities  including  those  at  Vandenberg  AFB. 
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(iii)  Logistics  Support  Analysis  (LSA)  which  developed  the  requirements  for 
sustainability  and  for  deliverable  test  equipment  for  both  the  depots  and  the  field 
sites. 

(iv)  Assembly  and  Check-Out  (A&CO)  Analysis  which  developed  the  A&CO 
requirements  on  the  prime  equipment  as  well  as  special  equipment  to  accomplish 
A&CO  in  the  field. 

Peacekeeper/4  Investing  in  long  term  systems  engineering  and  program  management  staff 
supports  program  success. 

Key  leaders  in  the  program  had  one  or  more  lengthy  BMO  assignments  prior  to  Peacekeeper 
development  or  had  other  in-depth  space  related  experience.  The  supporting  staff  had  similar 
experience  either  as  military  officers  with  multiple  space-related  assignments,  government 
civilians  with  lengthy  tenure  at  BMO  or  as  the  SE/TA  contractor  with  a  staff  of  well- 
experience  engineers,  scientists,  and  support  staff.  While  this  situation  was  not  unusual  at 
the  time,  it  seldom  occurs  in  today’s  program  offices. 

The  number  and  types  of  personnel  assigned  to  the  Peacekeeper  program;  high  levels  of 
knowledge,  skill,  ability,  and  experience  of  the  personnel,  and  lengthy  tenure  assignment  at 
BMO  all  contributed  tremendously  to  the  success  of  the  program.  These  aspects  reflect  the 
importance  of  the  Manpower,  Personnel,  and  Training  domains  associated  with  Human 
Systems  Integration  (HSI)  in  the  systems  engineering  process.  While  HSI  was  not  a 
common  term  at  the  time  of  Peacekeeper  development,  many  of  the  tenets  of  good  HSI  as  we 
know  it  today  were  applied,  particularly  in  these  domains. 

The  engineering  and  advanced  degrees  held  by  most  of  the  leadership  were  critical  in 
enabling  them  to  make  informed  decisions  and  manage  their  programs.  The  engineers  were 
allowed  sufficient  time  to  learn  missile  systems,  gain  experience  in  all  aspects  and 
subsystems,  and  learn  and  practice  the  BMO  Systems  Engineering  Process.  This  knowledge 
and  experience,  coupled  with  the  major  increase  in  manning  numbers  at  that  time,  primed  the 
program  for  success.  The  Air  Eorce  staff  had  the  manning  and  expertise  to  carefully  monitor 
subcontractors  and  actually  participate  in  detailed  reviews,  allowing  them  to  make  all  critical 
decisions  for  the  best  outcome  of  the  Air  Eorce. 

Adequate  manning,  experiential  learning,  supportive  work  environment,  hands-on 
experience,  job  rotation,  mentoring,  hand-picked  staff,  and  staff  education  level  (Manpower, 
Personnel,  and  Training  characteristics)  were  key  factors  in  the  success  of  the  Peacekeeper 
program. 

Peacekeeper/5  System  Requirements  Definition  must  be  completed  prior  to  development  to 
reduce  program  risk  and  cost. 

The  Peacekeeper  System  was  developed  to  meet  the  perceived  ICBM  “vulnerability  gap.”  A 
requirements  debate  centered  on  whether  the  Soviets  had  indeed  improved  accuracy  to  a 
point  that  silo-based  systems  were  vulnerable  to  a  first  strike.  The  Peacekeeper  provided  a 
very  highly  accurate  missile  with  ten  warheads  that  provided  increased  offensive 
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capability — but  could  they  be  deployed  in  a  mode  that  was  relatively  free  from  first  strike 
annihilation?  The  Peacekeeper  basing  mode  requirements  analyses  took  place  in  a  very 
public  forum  with  intense  input  from  non- traditional  DOD  sources.  Unlike  previous  weapon 
systems  (such  as  Minuteman)  that  developed  a  basing  mode  in  a  more  classified  environment 
with  little  public  debate,  the  Peacekeeper  system  was  studied  intensely  with  several  hundred 
different  modes  considered.  While  the  missile  requirements  were  essentially  completed  prior 
to  development,  the  basing  mode  changed  constantly  during  this  development  period.  While 
the  systems  engineering  analysis  considered  performance  and  survivability  merits,  a  major 
portion  of  the  analysis  focused  on  regional  and  national  economic  and  environmental 
considerations. 

This  Peacekeeper  case  study  revealed  the  importance  of  adhering  to  a  rigorous  systems 
requirements  analysis  (SRA)  process.  This  SRA  process  had  started  on  previous  programs 
at  the  BMO  and  produced  a  well  trained  engineering  force  that  transparently  used  the  SRA 
process  on  the  Peacekeeper  program.  The  study  also  demonstrated  the  importance  of  short 
and  long  term  technology  planning.  BMO  carefully  invested  resources  in  new  and  updated 
technologies  with  each  new  program,  but  only  inserted  them  when  they  were  technologically 
mature.  This  is  similar  to  today’s  Technology  Readiness  Level  (TRL)  analysis  and  the 
concept  of  spiral  development.  The  case  study  also  highlighted  the  importance  of  developing 
system  engineers  and  allowing  them  to  progress  through  a  variety  of  missile  programs  to  gain 
expertise.  Finally,  all  of  these  elements  combined  to  allow  BMO  to  successfully  develop, 
produce  and  deploy  a  weapon  system  despite  significant  public  debate  and  Congressional 
micro-management. 

A-10/1  The  system  concept  and  preliminary  design  must  follow,  not  precede,  the  mission 

analysis. 

The  A- 10  would  have  been  a  very  different  aircraft  had  this  principle  been  violated.  The 
clear  predilection  within  the  Air  Force  at  the  time  of  needs  definition  was  for  fast  multi¬ 
purpose  aircraft.  By  concentrating  on  the  Close  Air  Support  mission,  and  focusing  on  key 
characteristics  of  that  mission,  the  early  A-X  concept  working  group  was  able  to  discover 
what  the  critical  performance  parameters  were  that  contributed  to  these  characteristics.  An 
example  of  this  approach  is  how  the  group  treated  the  key  mission  characteristic  of 
responsiveness.  While  a  contributor  towards  responsiveness  can  obviously  be  aircraft  speed, 
the  group  understood  that  responsiveness  was  even  more  dependent  on  how  close  to  the 
battlefield  the  aircraft  could  be  based  and  maintained,  how  long  the  aircraft  could  loiter 
around  the  battlefield,  and  whether  or  not  the  pilot  could  easily  communicate  and  coordinate 
with  the  ground  troops  they  were  supporting.  The  aircraft  performance  parameters  were 
analyzed  in  terms  of  alternative  design  approaches  and  aircraft  design  parameters  in  areas  of 
airframe  and  propulsion,  avionics,  armament,  and  survivability  provisions.  Once  those 
design  parameters  were  understood  and  traceable  back  to  mission  characteristics,  the  study 
group  was  able  to  evaluate  candidate  aircraft  configurations  in  terms  of  mission  and  cost 
effectiveness.  This  front  end  application  of  Systems  Engineering  led  to  well  understood 
requirements  and  provided  a  solid  foundation  with  which  to  solicit  contractor  proposals  and 
start  a  development  program. 
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Part  of  the  A-X  concept  from  the  beginning  was  a  low  ownership  cost.  While  the  A- 10  did 
not  meet  its  intended  design-to-cost  goal,  the  cost  driven  approach  permeated  all  aspects  of 
the  design,  development  and  production  of  the  aircraft.  The  design  was  largely  constrained 
to  use  “existing  state-of-the-art”  technology,  avionics  were  kept  to  a  minimal  set  necessary  to 
accomplish  the  primary  missions,  and  the  design  incorporated  many  features  to  reduce  the 
maintenance  and  support  cost  for  the  aircraft.  An  example  of  this  is  the  attention  paid  to 
reducing  the  cost  associated  with  the  ammunition,  the  majority  driver  for  ownership  cost  of 
the  gun  system.  The  program  paid  more  in  the  development  effort  (due  to  multiple 
ammunition  subcontractors)  to  ensure  a  low  production  cost  would  be  attainable  when  it 
came  time  for  large  competitive  contracts  for  ammunition  to  support  operational  use. 


A- 10/2  Prototyping  can  be  used  to  help  manage  technical  and  cost  risk  at  the  system, 
pubsystem,  and  component  level. 

There  are  mixed  feelings  throughout  the  DoD  with  respect  to  prototyping,  and  in  particular 
competitive  prototype  phases  prior  to  full  scale  development.  Most  would  agree  that 
technical  and  life  cycle  cost  risk  can  be  partially  mitigated  by  delaying  costly  commitment 
decisions  for  full  scale  development  until  after  the  basic  design  has  been  demonstrated  and 
cost  estimates  have  matured.  The  typical  downside  of  this  approach  is  that  development 
costs  rise  and  development  schedules  lengthen,  and  clearly  this  is  what  occurred  as  a  result  of 
the  decision  to  use  the  competitive  prototype  phase  for  the  A-X  program.  Some  would  argue 
that  the  additional  development  expense  of  these  competitive  prototype  phases  makes  them 
impractical  for  large  systems  (e.g.,  bomber  aircraft,  ships).  An  example  that  can  be  taken 
away  from  the  A- 10  case  study  is  how  this  can  be  done  at  the  subsystem  and  component 
level.  The  gun  system  on  the  A- 10  was  considered  to  be  risky,  so  a  competitive  prototype 
phase  for  the  GAU-8/A  program  was  run  in  parallel  with  that  for  the  A-X  aircraft.  Further, 
the  30  mm  ammunition  was  assessed  as  having  both  technical  and  cost  risk,  so  competitive 
prototyping  was  used  at  the  subcontractor  level  within  the  GAU-8/A  program.  With  the 
expected  ownership  cost  of  the  gun  being  driven  by  the  ammunition  cost,  the  intent  was  to 
have  multiple  qualified  sources  for  ammunition  by  1978  when  a  direct  competitive  contract 
would  be  let  to  procure  ammunition  for  operational  use.  While  the  GAU-8/A  program  paid 
the  development  cost  for  this  risk  mitigation  approach,  a  good  gun  system  with  relatively  low 
operating  cost  was  obtained  as  a  result. 

^-10/3  Clear  lines  of  responsibility  must  be  established  to  ensure  successful  integation, 
bspecially  when  multiple  progams  are  involved. 

The  Air  Force  and  the  program  office  understood  what  the  A- 10  system  integration  risks 
were,  and  took  steps  to  mitigate  them.  These  risks  were  associated  with  the  integration  of  a 
large  caliber  internal  gun  system,  and  the  ammunition  associated  with  the  gun.  The  aircraft, 
the  gun,  and  the  ammunition  were  all  parallel  development  efforts,  and  it  was  important  that 
they  all  came  together  if  the  A-10  was  to  provide  the  capability  the  Air  Force  needed.  The 
Air  Force  addressed  this  integration  risk  directly  by  how  they  set  up  responsibility  for  the 
various  programs.  Managerial  responsibility  for  the  GAU-8/A  gun  system  was  given  to  the 
A-10  program  office,  and  the  memorandum  of  understanding  establishing  this  relation 
between  the  aircraft  and  gun  program  was  later  expanded  to  include  the  ammunition  as  well. 
The  ammunition  suppliers  were  established  under  a  subcontract  relation  to  GE,  the  prime 
contractor  for  the  gun  system.  This  made  GE  responsible  for  the  total  gun  system  and  a 
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single  program  office  responsible  for  the  combined  aircraft  and  gun  system.  While  the  Air 
Force  did  not  establish  this  same  type  of  relationship  with  the  engine  program,  the  TF-34 
engine  was  already  under  development  for  another  aircraft  at  the  time  the  A-X  program 
began.  Further,  the  engine  placement  off  the  fuselage  of  the  A- 10  allowed  for  relatively  easy 
integration  with  the  airframe. 


A- 10/4  The  government  must  ensure  the  contractor  is  able  to  “Walk  the  Talk”  when  it  comes 
to  production. 

In  reviewing  the  abilities  of  the  contractor  to  execute  the  contract,  the  Air  Force  failed  to 
identify  a  number  of  issues  that  might  well  have  doomed  the  program  to  failure.  Both  before 
and  after  the  awarding  of  the  contract  the  company  was  in  trouble.  The  government’s  pre¬ 
award  survey  of  Fairchild  Hiller  examined  their  capacity,  capability  and  financial  condition, 
but  failed  to  recognize  some  of  the  risk  elements  and  concerns  that  would  be  noted  some  3 
years  later.  Fairchild  had  failed  to  adequately  invest  in  equipment  and  its  workforce,  and  its 
management  and  organization  had  deficiencies.  The  Air  Force  made  a  number  of 
recommendations  that  were  followed,  and  the  company  was  able  to  produce  the  aircraft; 
however,  it  ultimately  put  the  company  in  a  position  from  which  it  could  not  recover. 

A- 10/5  Successful  design,  development  and  production  is  not  enough  to  sustain  a  system 
throughout  its  life  cycle. 

By  most  accounts,  the  A- 10  was  a  well  designed  and  well  built  aircraft.  Its  performance  in 
Desert  Storm  certainly  supports  this  claim.  Prior  to  Desert  Storm,  however,  the  Air  Force 
had  committed  to  retiring  the  A- 10  and  this  adversely  affected  the  sustainment  efforts  for  the 
aircraft.  While  the  Air  Force  reversed  the  decision  to  retire  the  A-10  shortly  after  Desert 
Storm,  it  did  not  make  the  investments  in  maintenance  and  modifications  that  should  have 
followed  that  reversal.  Multiple  sources  noted  systemic  neglect  of  the  inspection, 
maintenance,  and  attention  to  service  life  issues  throughout  the  late  1980’s  and  most  of  the 
1990’s.  Required  inspections  were  not  conducted,  early  fatigue  indications  at  Wing  Station 
23  and  other  locations  were  effectively  ignored,  and  critical  data  regarding  actual  service  life 
was  no  longer  being  obtained.  This  was  further  impacted  by  major  disruptions  to  the 
sustainment  program  office  as  a  result  of  the  1995  Base  Realignment  and  Closure  decisions. 
The  program  office  reported  80%  loss  of  experienced  personnel  due  to  the  work  transfer 
from  McClellan  to  Hill  AFB,  and  repeated  turnover  at  the  System  Program  Director  and 
Chief  Engineer  level  was  equally  problematic.  The  Air  Force  lost  awareness  of  the  structural 
health  of  the  A-10  fleet,  and  the  intended  repair  for  the  aircraft,  which  did  not  address  the  full 
scope  of  issues  required  to  provide  the  required  life  extension,  proved  to  be  costly  and 
ineffective  in  providing  the  required  service  life.  The  end  result  was  a  decision  to 
manufacture  new  wings  for  all  thin  skin  aircraft  remaining  in  the  fleet.  Had  the  Air  Force 
maintained  awareness  of  the  structural  health  of  the  A-10,  the  decision  to  re -wing  could  have 
been  made  earlier,  possibly  obviating  the  need  for  the  wing  components  of  the  HOG  UP 
program.  This  likely  would  have  resulted  in  an  overall  cost  savings  and  earlier  operational 
capability  for  the  re-winged  aircraft. 
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A- 10/6  “If  the  politics  don’t  fly,  the  sgfstem  never  will.”* 


The  quote  defining  this  Learning  Prineiple  is  one  of  the  heuristies  from  the  text  The  Art  of 
Systems  Architecting  by  Mark  Maier  and  Eberhart  Reehtin.  In  that  text  the  authors  devote  an 
entire  chapter  to  “The  Political  Process  and  Systems  Architecting”.  Other  heuristics  from 
that  chapter  include; 

•  Affordability  is  decided  by  whichever  side  has  the  most  votes; 

•  A  strong,  coherent  constituency  is  essential; 

•  Technical  problems  become  political  problems; 

•  The  best  engineering  solutions  are  not  necessarily  the  best  political  solutions. 

All  of  these  heuristics  can  be  applied  in  retrospect  to  the  A- 10  as  it  was  beset  with  political 
fighting  for  the  entire  life  of  the  program.  Early  on,  internal  Air  Eorce  politics  made  it 
difficult  to  pursue  a  specialized  CAS  aircraft  when  the  majority  of  the  service  favored  fast 
multi-purpose  fighters.  Later,  it  faced  difficulties  justifying  its  existence  alongside  the  Army 
attack  helicopters;  however,  it  may  have  been  the  very  existence  of  these  advanced  helicopter 
development  programs  that  allowed  it  to  overcome  resistance  to  the  CAS  aircraft  within  the 
Air  Eorce.  Still  later,  the  A- 10  was  forced  to  walk  a  tight  line  between  those  that  wanted  a 
simpler,  lower  cost  aircraft,  and  those  that  didn’t  want  the  A- 10  because  of  its  shortcomings 
at  night  and  in  adverse  weather  (brought  about  by  a  “lean”  avionics  package  to  keep  costs 
down).  Politics  shaped  the  A- 10  program  structure,  most  notably  in  the  adoption  of  a 
politically  attractive  competitive  prototyping  phase  associated  with  both  the  A-X  aircraft  and 
the  GAU-8/A  gun.  Congressional  politics  forced  an  additional  competition  with  the  A-7D 
even  after  the  competitive  fly-off  between  the  YA-9  and  YA-10,  and  attempted  to  force 
another  show  down  with  the  Piper  Enforcer.  Einally,  congressional  politics  attempted  to 
derail  redistribution  of  production  required  to  address  deficiencies  associated  with  the 
Earmingdale  NY  plant.  The  proponents  of  the  A- 10  won  some  of  these  battles,  and  lost  some 
as  well,  but  persevered  in  cobbling  together  the  “strong,  coherent  constituency”  considered 
essential  to  the  success  of  the  program. 

GH/1  While  an  “evolving  requirements”  strategy  may  be  an  excellent  choice  for  a  concept 
demonstration  program,  it  is  an  unwise  strategy  for  an  EMD  program. 

Typically,  a  System  Specification  derived  from  the  SRD  is  placed  on  contract  at  the 
beginning  of  EMD  to  establish  the  contractual  basis  for  the  subsequent  design  and 
verification  activities  and  support  the  definition  of  the  scope  and  cost  of  the  development 
program.  With  the  spiral  development  approach  that  was  employed,  there  was  no  contractual 
System  Specification  at  the  start  of  Spiral  1  or  Spiral  2  (each  of  which  constituted  significant 
EMD  programs),  just  a  draft  SRD  that  contained  top-level  requirements.  The  intent  was  to 
evolve  to  the  baseline  technical  requirements  at  the  completion  of  EMD,  thus  providing  the 
contractor  with  greater  contractual  latitude,  similar  to  the  ACTD  program.  However,  at  this 
point  in  the  program,  the  user  had  specific  performance  requirements  that  needed  to  be  met, 
and  these  were  not  necessarily  consistent  with  available  funding.  The  result  was  a  choice  of 
“the  devil’s  alternative”:  either  breach  the  program  cost  ceiling  or  fail  to  meet  the  user 


*  Maier,  Mark  &  Reehtin,  Eberhardt,  The  Art  of  Systems  Architecting,  2"‘‘  ed.,  CRC  Press,  New  York,  2002. 
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requirements.  Not  having  a  firm  set  of  technical  requirements  also  increased  the  program  risk 
relative  to  requirements  creep  and  baseline  verification  of  engineering  content.  As  a  result, 
the  program  experienced  a  Nunn-McCurdy  cost  breach  in  excess  of  the  25  percent  threshold 
during  Spiral  2  (Block  20)  of  EMD. 

GH/2  An  overall  systems  engineering  strategy  must  be  defined  as  early  as  possible  in  ij 
progiram  and  must  be  consistent  with  the  acquisition  abroach  and  progiam  needs. 

During  the  ACTD  phase,  the  contractor  had  a  small  systems  engineering  organization 
consisting  of  systems  engineers  located  within  each  of  the  Integrated  Product  Teams  (IPTs). 
The  contractor  integrated  the  systems  engineering  function  partly  into  the  technical 
management  structure  wherein  some  of  the  IPT  leads  were  experienced  in  the  systems 
engineering  discipline.  The  Chief  Engineer  created  this  organization  and  chose  the  IPT  leads 
based  on  leadership  capability,  interpersonal  skills,  and  technical  excellence  in  their  area  of 
expertise  (systems  engineering,  design,  analysis,  manufacturing/assembly,  logistics,  cost, 
ground  segment,  communications,  payloads,  etc.)  In  this  concept,  the  Chief  Engineer  was  the 
individual  with  primary  responsibility  to  define  and  execute  the  systems  engineering  process. 
At  the  start  of  EMD,  the  systems  engineering  function  was  largely  integrated  into  the  IPTs. 
About  two  years  into  EMD,  the  contractor  reorganized,  and  the  systems  engineers  were 
centralized  into  a  Systems  Engineering  Integration  Team  (SEIT).  The  SEIT  was  viewed  by 
the  IPTs  as  a  “non-value  added”  activity  and  “watch  dog”  that  imposed  workload  that 
detracted  from  the  IPT’s  ability  to  accomplish  required  tasks.  Consequently,  the  SEIT  had 
very  limited  ability  to  successfully  impact  the  conduct  of  the  program.  This  is  supported  by 
the  point  that,  when  funding  problems  drove  program  cutbacks,  systems  engineering  proved 
to  be  an  easy  target.  As  a  result,  the  Spiral  2  effort  was  lacking  in  processes  necessary  to 
support  the  efficient  completion  of  an  EMD  program. 

OH/3  The  more  an  acquisition  strategy  plans  concurrency  between  developing  the  validate^ 
baseline  and  production,  the  hi;^er  the  risk  of  future  cost  increases  and  schedule  delays. 

Before  a  program  proceeds  into  production,  there  is  a  need  that  all  participants  understand 
how  the  product  baseline,  expressed  in  terms  of  performance  capabilities  and  limitations, 
meets  the  functional  baseline,  expressed  in  terms  of  technical  requirements.  If  this  mutual 
understanding  does  not  exist,  then  there  is  a  significant  risk  that  the  user  needs  will  not  be 
met. 

OH/4  The  more  complex  the  program,  the  more  critical  it  is  that  an  IMS  links  all  aspects  of 
the  program,  including  the  lower-tiered  schedules. 

The  Spiral  1  EMD  Statement  of  Work  (SOW)  required  that  the  contractor  create  and 
maintain  an  IMS  that  showed  interdependencies  and  critical  paths.  The  contractor  responded 
by  creating  a  multitude  of  IMSs:  one  for  each  Spiral  and  one  for  each  second  tier  IPT.  The 
IMSs  were  independent  of  each  other,  did  not  address  subcontracted  efforts,  and  there  was  no 
single  overarching  IMS  that  integrated  all  the  lower-tiered  IMSs.  The  Global  Hawk  program 
was  complex  with  multiple  spirals  occurring  simultaneously,  and  few  individuals,  if  any, 
knew  how  all  the  “pieces”  fit  together.  The  lack  of  a  single,  integrated  schedule  showing 
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overall  program  critical  paths  and  interdependencies  made  it  impossible  for  program 
management  to  truly  assess  the  overall  program  status.  In  reality,  no  one  knew  how  one 
schedule  slip  impacted  another  and  whether  someone  else’s  critical  path  was  being  violated. 


QH/5  Contractual  programmatic  and  technical  requirements  must  be  clearly  defined, 
understood,  and  executed  by  all  parties  to  include  the  areas  of  airworthiness  certification  and! 
Boftware  development  and  verification. 

In  order  to  avoid  disputes  between  the  contractor  and  Air  Force  customer,  it  is  necessary  to 
contractually  define  key  programmatic  and  technical  requirements.  The  contractual  wording 
needs  to  be  clear  and  unambiguous.  Also,  particular  care  needs  to  be  taken  to  ensure  that  all 
parties  share  the  same  understanding  of  the  requirements  and  that  the  requirements  are 
executed  in  a  timely  fashion.  This  principle  was  violated  in  two  key  areas:  airworthiness 
certification  and  software  development/validation. 

QH/6  Development  testing  must  follow  a  disciplined  engineering  process  or  cost,  schedule, 
and  technical  risks  may  be  incurred. 

The  OSD  assessment  in  support  of  the  Nunn-McCurdy  recertification  concluded  that  the 
program  development,  test,  and  evaluation  (DT&E)  strategy  was  insufficient  to  reduce 
program  risk  and  support  knowledge-based  decisions.  Lower-level  development  tests  were 
typically  not  used  as  a  building  block  to  system  test,  entry  and  exit  criteria  were  typically 
missing,  test  planning  and  test  procedures  were  inadequate,  critical  test  parameters  were 
often  not  verified,  test  assets  were  lacking,  and  systems  were  not  assessed  against  their 
intended  environment.  In  short,  OSD  concluded  that  the  lack  of  a  disciplined  engineering 
approach  to  development  testing  was  a  contributor  to  the  Nunn-McCurdy  breach. 


KC  -135/1  Systems  Engineering  must  translate  program  goals  and  objectives  into  clearly 
defined  and  verifiable  system  requirements,  focusing  on  the  entire  life  cycle. 

Key  elements  of  requirements  allocation  include  early  involvement  by  the  support  contractor 
with  the  aircraft  systems  prime  contractor(s)  to  ensure  simulator  specific  requirements  are 
addressed.  Because  aircraft  upgrades  are  identified  by  the  KC-135  Program  Office  at  Tinker 
AFB  there  are  roadmap  meetings  held  at  Tinker  where  upcoming  modifications  to  the  KC- 
135  aircraft  are  discussed.  The  KC-135  ATS  O&M  contractor  and  the  Training  System 
Program  Office  at  Ogden  now  are  present  to  assess  those  modifications  and  ensure  the  ATS 
requirements  are  included  in  the  early  planning  process. 


KC- 135/2  The  Systems  Engineering  process  must  be  structured  to  properly  mitigate 
challenges  generated  by  third-party  Modification  Contractors. 

The  KC-135  O&M  contract  states  the  prime  contractor,  ElightSafety,  is  responsible  for 
meeting  the  overall  performance  requirements  of  the  training  system  including  trained 
students  that  meet  Government  standards.  Competitive  contracting  for  early  OET  upgrades 
did  not  formally  involve  buy-in  from  the  O&M  contractor  who  retains  ultimate  responsibility 
for  providing  a  “guaranteed”  student  to  the  Government.  This  was  recognized  and  remedied 
in  future  contracts  by  involving  the  support  contractor  in  a  more  disciplined  manner  to  ensure 
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early  involvement  in  the  development  effort.  For  example,  the  requirement  for  a  Performance 
Work  Statement  (PWS)  has  evolved  that  identifies  specific  tasks  to  be  accomplished  by  the 
KC-135  ATS  prime  in  order  to  ensure  training  system  requirements  are  identified  at  the  ATS 
level  and  properly  allocated  to  the  various  subsystems  under  development. 


KC- 135/3  Systems  engineers  must  be  responsible  for  ensuring  that  all  stakeholders  aw 
pivolved  during  key  decision  technical  planning  and  execution  process  reviews. 

The  KC-135  program  requires  a  tailored  set  of  formal  reviews  be  held  during  the 
development  phase  that  is  based  on  the  size  and  complexity  of  the  modification  program. 
These  reviews  ensure  that  the  entire  KC-135  ATS  team  is  working  to  the  same  requirements, 
designing  and  developing  the  correct  modifications,  adequately  testing  the  modifications  and 
generating  the  appropriate  courseware  changes.  The  reviews  employed  throughout  the 
modification  development  and  verification  efforts  may  include  a  Systems  Requirements 
Review  (SRR),  Preliminary  Design  Review  (PDR),  Critical  Design  Review  (CDR),  Test 
Readiness  Review  (TRR),  Required  Assets  Available  Review  (RAAR)  and  In  Process 
Reviews  (IPR).  KC-135  simulator  reviews  are  structured  to  ensure  that  the  KC-135 
Simulator  team  has  mutual  expectations  and  understanding  of  requirements  and  that  the 
contractor’s  proposed  preliminary  designs  and  program  plan  satisfies  the  development 
specification. 

KC- 135/4  Integrated  logistics/maintainability  sugnort  structure  avoids  parts  obsolescence  anj 
diminishing  manufacturer  su^ly  issues. 

As  with  aircraft  systems,  training  systems  must  also  have  detailed  technical  plans  that 
integrate  a  logistics/maintainability  support  structure  to  ensure  continued  future  operation. 
During  each  technical  planning  activity,  systems  engineering  must  be  concerned  with  the 
total  support  of  the  system  to  assure  its  economic  and  effective  operation  throughout  its  life 
cycle.  Logistics  objectives  for  the  program  need  to  be  included  in  these  technical  plans  to 
ensure  achieving  stated  readiness  objectives  such  as  system  availability,  programmed  flying 
training  throughput,  establishment  of  Reliability  and  Maintainability  performance 
requirements  needed  to  support  readiness  objectives,  and  emphasizing  logistics  support 
considerations  in  all  design  trade  studies. 


KC  -135/5  Simulator  modeling  data/modification  requires  verification  and  validation  to| 
ensure  aircraft-like  flying  qualities. 

AMC  and  FlightSafety  personnel  have  also  developed  a  test  and  evaluation  process  that 
promotes  confidence  that  KC-135  simulator  modifications  “fly”  like  the  KC-135  aircraft. 
Even  though  the  systems  integrating  contractor  and  the  Government  quantitatively  specify 
many  requirements,  the  final  evaluation  of  the  training  realism  is  still  subjectively  validated. 
In  the  end,  the  trainer  model  must  be  correct  enough  to  allow  training,  which  means  it’s  a 
judgment  call  by  the  test  crews  and  Air  Force  instructors  about  the  system’s  “training  value.” 
The  KC-135  ATS  team  has  implemented  an  approach,  which  utilizes  no  more  than  one  or 
two  contractor  instructor  (FlightSafety)  pilots  and  one  Air  Force  instructor  pilot  to  minimize 
extended  test  periods  and  facilitate  reaching  consensus  on  a  modification’s  training  value. 
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